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Guest  Editorial 


“Plug  and  Test”:  The  Goal  of  Distributed  T&E 

Michael  D.  Crisp 

Office  of  the  Director,  Operational  Test  and  Evaluation  (DOT&E), 

Office  of  the  Secretary  of  Defense  (OSD),  Washington,  D.C. 


Just  over  two  years  ago,  I  took  on  the  challenge  of 
leading  the  development  of  a  Roadmap  to  help 
steer  the  Department  of  Defense  (DoD)  toward 
effectively  conducting  testing  in  a  joint  environ¬ 
ment.  DoD  had  figured  out  that 
military  systems,  even  so-called  “simple” 
Service-centric  systems,  were  eventually 
going  to  operate  in  a  much  larger  joint  and 
coalition  environment.  The  Global  War  on 
Terror  has  forced  a  change  in  DoD,  with  a 
realization  that  we  needed  to  figure  out 
how  systems  play  together  and  their  contri¬ 
bution  to  mission  outcome.  To  win  the 
Global  War  on  Terror,  the  consensus  with¬ 
in  DoD  is  to  be  joint — in  warfighting, 
training  and  acquisition.  This  means  that 
we  must  “conceive  joint”  capabilities,  devel¬ 
op  “born  joint”  material  solutions  and  “test  as  we  fight”  in 
a  realistic  environment.  The  “Testing  in  a  Joint 
Environment  Roadmap”  was  approved  by  the  Deputy 
Secretary  of  Defense  in  November  2004,  and  set  DoD  on 
a  course  for  joint  experimentation,  acquisition,  training 
and  testing.  As  a  formal  program  of  record,  the  Joint 
Mission  Environment  Test  Capability  (JMETC)  is  a  net- 
centric  enabler  for  conducting  joint  systems  demonstra¬ 
tions  across  the  experimentation,  engineering,  testing  and 
training  domains. 

The  Roadmap 

The  Testing  in  a  Joint  Environment  Roadmap  pro¬ 
posed  changes  that  will  enable  the  test  and  evaluation 
(T&E)  community  to  “test  like  we  fight.”  The  Roadmap 
promotes: 

■  Institutionalizing  the  need  to  test  in  realistic  joint 
operational  environments.  The  Roadmap  requires  changes 
to  DoD  policy  and  enforcement  by  leadership. 

■  Defining  capabilities  in  common,  measurable 
warfighting  terms — an  essential  element  in  establishing 
an  evaluation  continuum  over  the  lifecycle  of  systems. 

■  Establishing  persistent  connectivity  between  battle 
labs,  hardware-in-the-loop  simulations,  developmental 
test  facilities  and  live  force  instrumentation.  This  is  neces¬ 
sary  to  achieve  net-centricity  and  interoperability. 

■  Using  this  persistent  connectivity  to  achieve  robust 
live-virtual-constructive  (LVC)  joint  mission  environ¬ 


ments  for  joint  experimentation,  development,  test  and 
training.  Persistent  connectivity,  with  standardized 
processes,  protocols  and  interfaces,  is  needed  to  make 
comparable  the  results  of  these  communities. 

As  a  step  forward  to  conducting  real¬ 
istic  and  adequate  joint  testing,  the 
Roadmap  also  recommended  that  DoD 
should: 

■  Share  test  and  Joint  National  Training 
Capability  venues  and  resources. 

■  Allow  for  increased  use  of  the  Guard 
and  Reserve  forces,  where  appropriate. 

■  Revitalize  modeling  and  simulation 
to  achieve  the  DoD  vision  of  a  decade  ago. 

Taken  as  a  whole,  the  Roadmap  is  an 
important  enabler  for  acquiring  capabili¬ 
ties  that  are  “born  joint,"  and  testing 
legacy  equipment  and  systems  that  are  “made  joint.” 
The  Secretary’s  guidance  established  new  DoD  policy 
to  be  institutionalized ,  that  we  will  conduct  testing  in  a 
joint  environment  where  applicable,  and  directs  that  we 
provide  the  resources  required.  The  stage  has  been  set,  and 
the  process  for  joint  testing  has  begun. 

Understanding  distributed  T&E 

Distributed  T&E  and  its  place  within  testing  in  a  joint 
environment  means  different  things  to  different  people, 
depending  on  the  type  of  technology  at  play.  Distributed 
T&E  provides  the  ability  to  demonstrate  system  perform¬ 
ance  capabilities  by  remotely  interfacing  other  system  ele¬ 
ments,  their  stimuli ,  and  users  by  forming  stable,  repeat- 
able,  dynamic  and  realistic  joint  mission  environments. 
The  system  elements  can  be  LVC  or  any  combination 
thereof.  Distributed  T&E  can  be  applied  throughout  the 
development  cycle  from  requirements  generation, 
through  design,  engineering,  product  acceptance  and  in- 
Service  support.  It  is  not  a  new  environment,  require¬ 
ment,  phase  or  method  of  testing,  but  rather,  a  more  effi¬ 
cient  and  effective  way  of  doing  what  we  have  always 
done.  Instead  of  bringing  in  all  interfacing  elements  to  the 
test  range,  we  merely  “plug  and  test”  into  the  T&E  “net,” 
so  to  speak.  The  idea  is  not  new,  as  the  commercial 
telecommunications  industry  long  ago  figured  out  that 
distributed  operations  naturally  demanded  distributed 
performance  monitoring.  It  makes  little  programmatic 
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sense,  much  less  a  sound  business  case,  to  separately  fund 
program-specific  test  networks  that  are  highly  duplica¬ 
tive.  A  common,  distributed  test  capability,  as  represent¬ 
ed  by  JMETC,  will  provide  the  necessary  joint  context  for 
DoD  and  industry. 

Application  to  T&E 

Distributed  engineering  has  made  significant  strides  in 
the  DoD,  but  its  use  in  T&E  is  only  now  just  being  real¬ 
ized.  The  Navy  had  great  success  using  its  distributed 
engineering  capability  to  solve  vexing  combat  system 
problems  experienced  at  sea,  but  did  not  really  apply  it  to 
new  programs  under  development.  The  Air  Force  more 
efficiently  trained  pilots  and  analyzed  platform  contribu¬ 
tion  to  mission  outcome  through  distributed  air  combat 
simulations,  but  used  such  capability  only  in  unique  cases 
for  T&E  of  weapons.  For  more  than  15  years,  the  Army 
used  its  distributed  engineering  infrastructure  to  develop 
requirements  for  future  systems,  but  never  much  for  actu¬ 
al  program  engineering  and  testing. 

It  is  only  in  the  last  few  years  that  these  capabilities  have 
become  integral  aspects  of  the  system  development  itself. 
Our  systems  have  become  so  complex,  with  systems  inte¬ 
gration  such  a  driving  element  of  acquisition,  that  I  can 
safely  say  that  today,  all  of  our  most  complex  systems  under 
development  have  been  forced  to  build  distributed  engi¬ 
neering  and  test  capabilities... unfortunately,  true  to  past 
paradigms,  most  solely  for  their  own  use.  Programs  confi¬ 
dently  assert  that  they  can  now  “plug  and  test,”  but  for  the 
most  part,  they  can  do  so  only  within  their  own  mission 
arena  under  specific  program  applications.  Unfortunately, 
we  continue  to  be  very  adept  at  ensuring  that  every  garage 
in  the  neighborhood  has  its  own  proprietary  approach 
toward  distributed  systems  testing.  The  problem  is  that  we 
just  cannot  afford  to  do  business  this  way. 

Why  now? 

For  the  most  part,  the  T&E  infrastructure  that  contin¬ 
ues  to  serve  the  best  military  in  the  world  was  built  to 
assess  functionality  within  and  between  elements  of  that 
system,  primarily  on  a  program-by-program  basis.  As  sys¬ 
tem  functionality  grew,  programs  had  to  bear  the  cost  of 
adding  more  test  elements  to  their  infrastructure.  For  crit¬ 
ical  capabilities,  we  even  went  so  far  as  to  build  parallel  sys¬ 
tems  for  the  primary  purpose  of  testing.  We  were  com¬ 
fortable  in  our  own  little  controlled  T&E  worlds  as  we 
watched  our  tool  boxes  grow. 

Times  have  changed  and,  in  some  ways,  our  infrastruc¬ 
ture  was,  and  to  some  degree  still  is,  slow  to  respond.  The 
concept  of  net-centricity  introduced  government  and 


industry  to  system  interdependency  so  that  each  system 
element  capitalized  on  each  other’s  contribution  to  produce 
much  greater  overall  effect.  Systems  no  longer  had  real  con¬ 
trol  over  who  they  interfaced  with.  Their  job  was  to  push 
information  up,  without  necessarily  seeing  what  was  going 
to  be  done  with  it.  Small  programs  could  now  impact  big 
systems  in  big  ways.  We  accepted  this  “net-centricity”  as  a 
characteristic  delivered  to  the  user,  but  winced  when  apply¬ 
ing  it  to  engineering,  much  less  testing.  Why  did  the  prem¬ 
ise  of  distributed  testing  upset  our  paradigm  of  T&E  so 
much  that  it  became  acceptable  for  each  program  to  build, 
use  and  ultimately  tear  down  a  T&E  capability,  rather  than 
trust  entities  outside  of  their  immediate  control? 

A  new  vector 

The  Roadmap  set  a  clear  vector  for  DoD,  attempting 
to  mimic  what  private  industry  already  knows.  Keep  it 
lean,  and  capitalize  on  what  others  have  done  or  own  to 
lower  the  cost  of  bringing  goods  and  services  to  market. 
Fiscal  realities,  system  complexity  and  interdependency  of 
today’s  systems  tell  us  that  there  must  be  a  better  way  than 
having  programs  build  their  own  engineering  and  T&E 
infrastructure  only  to  see  it  dismantled  at  the  end  of  devel¬ 
opment.  Neither  does  DoD  need  another  new  “compli¬ 
ance”  site  to  verify  some  abstract  degree  of  “jointness”  or 
another  new  formal  test  phase  at  the  back  end  of  develop¬ 
ment.  The  Roadmap  lays  out  a  coherent  path  to  lash 
together  the  robust  capabilities  that  already  exist  within 
and  outside  of  DoD  for  the  primary  use  of  developers, 
testers  and  trainers.  Systems  are  both  users  and  contribu¬ 
tors  to  the  network.  The  Roadmap  is  not  just  some  new 
centrally  controlled  capability,  but  rather,  a  federation  of 
capabilities  under  the  Services’  control  for  use  by  everyone! 
JMETC  is  a  corporate  solution — it  serves  the  users  by  bring¬ 
ing  standardized  business  rules,  processes,  protocols  and  proce¬ 
dures  that  are  fungible  across  DoD  and  industry.  This  is  truly 
a  different  paradigm.  This  is  not  about  ownership,  but  of 
collaborative  participation.  We  can  no  more  centrally  own 
or  control  the  vast  T&E  infrastructure  resident  within 
DoD,  the  Services,  industry  and  academia  than  we  can 
own  or  control  the  Internet.  We  have  to  focus  on  better 
ways  to  plug  and  play  across  community  boundaries. 

Seeing  the  potential 

Lashing  together  such  a  capability  is  not  merely  aca¬ 
demic;  it  makes  good  business  sense  and  could  very  well  be 
the  key  to  freeing  us  from  oppressively  long  development 
cycles.  Making  it  easier  to  borrow  your  neighbor’s  saw  for 
awhile  is  certainly  more  cost  effective  than  buying  your 
own.  Imagine  the  potential  savings  for  a  missile  developer 
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where  engineers  from  the  seeker  division  merely  plug 
brassboard  elements  into  the  net  and  choose  from  a  menu 
of  simulated  and  constructed  stimuli  from  other  divisions 
(or  government  archives)  across  the  country  or  around  the 
globe.  Imagine,  instead  of  distributed  events  between  divi¬ 
sions  in  one  company,  we  have  a  dynamic  plug  and  test 
capability  between  industry,  government  and  academia. 
Users  act  as  customers,  selecting  the  elements  of  their  test 
environment  based  on  defining  and  understanding 
requirements,  cost,  pedigree  and  availability. 

Small  programs,  which  would  otherwise  defer  testing 
with  major  integrating  elements,  have  the  chance  to  play 
early  in  their  development  at  a  much  lower  cost  than  if  they 
pursued  it  on  their  own.  Imagine  a  robust  backbone  of 
T&E  capability  similar  to  our  interstate  highway  system, 
serving  not  only  those  myriad  customers  on  short,  point- 
to-point  trips  (for  example,  short-duration  engineering 
efforts  between  two  contractors  solving  a  mutual  interface 
problem),  but  at  the  same  time  those  on  more  dedicated 
long-haul  efforts  (such  as  participating  in  a  major  orches¬ 
trated  joint  Service  demonstration).  Imagine  being  able  to 
tap  into  live  sensors  in  far  away  theaters  of  conflict  and  feed 
them  into  centers  where  warriors  are  assessing  combat  sys¬ 
tems  still  under  development.  This  is  reality  today,  but  it 
has  to  be  done  at  the  corporate  level — in  a  persistent,  effi¬ 
cient  and  cost-conscious  manner.  As  system  capability 
requirements  and  interdependency  grow,  so  too  will  the 
environment  in  which  system  performance  and  contribu¬ 
tion  have  to  be  demonstrated.  A  robust,  distributed  “plug 
and  test”  capability  can  realistically  get  us  there. 

21st-century  solutions 

Distributed  engineering,  T&E  and  training  solutions 
are  already  a  reality,  but  they  are  not  focused  at  the  corpo¬ 
rate  level.  The  question  is  whether  we  can  achieve  consen¬ 
sus  and  a  common  business  approach  to  move  the  T&E 
community  toward  open  partnerships  and  collaboration 
across  the  joint  domain.  Today,  there  are  more  than  40  dis¬ 
tributed  engineering  and  test  networks  just  within  DoD. 
Program  managers  are  rarely  rewarded  for  diverting  scarce 
resources  for  investing  in  what  might  be  a  noble  effort  for 
the  common  good,  because  delivering  on-time  and  within 
budget  are  top  priorities.  The  reality  today  is  that  the 
investments  to  make  this  happen  are  already  being  made 
and  multiplied  many  times  over — no  new  real  investment  is 
necessary.  Most  programs  are  in  a  constant  state  of  devel¬ 
opment,  spiraling  out  products  on  a  periodic  basis  while 
providing  lifecycle  support  for  fielded  systems.  For  all 
intents  and  purposes,  some  degree  of  system  emulation,  be 
it  digital  or  actual  hardware-in-the-loop,  is  always  up  and 


running.  When  this  is  multiplied  by  almost  a  thousand 
systems  in  existence,  one  can  see  the  lost  opportunities  to 
reduce  excess  T&E  capabilities  and  program  costs. 
JMETC  is  a  first  step  in  creating  the  joint  mission  envi¬ 
ronment,  enabling  joint  solutions  for  the  warfighter. 

There  are  myriad  process,  facilities,  policy  and  propri¬ 
etary  issues  that  we  are  working  to  overcome,  but  these 
are  merely  sidebars  of  a  capability  that  is  already  here  and 
expanding  every  day.  We  as  a  community  have  to  embrace 
the  potential  of  distributed  T&E,  just  as  the  commercial 
sector  has  done,  and  set  aside  fears  of  losing  control  and 
discipline.  Plug  and  test  is  here,  and  it  works.  □ 

MICHAEL  D.  CRISP  serves  as  the  deputy  director.  Air 
Warfare,  Operational  Test  and  Evaluation  (OT&E),  Office  of 
the  Secretary  of  Defense  (OSD),  Washington,  D.C.,  where  he 
is  directly  responsible  for  the  adequacy  of  OT&E  for  air  sys¬ 
tems  within  the  Department  of  Defense  (DoD).  His  responsi¬ 
bilities  include  oversight for  the  conduct  of  major  weapons  sys¬ 
tems  assessments  that  are  reported  to  the  Secretary  of  Defense 
and  Congress,  and  to  support  the  participation  of  the  Director, 
OT&E,  on  the  Defense  Acquisition  Board  and  in  the  Defense 
Acquisition  Executive  Summary  process.  He  also  serves  as  pro¬ 
gram  director  for  the  Joint  Test  and  Evaluation  program, 
which  funds  joint  Service  projects  to  develop  joint  tactics,  tech¬ 
niques  and  procedures  to  better  support  joint  operations.  He  also 
oversees  efforts  to  develop  and  implement  the  DoD  Joint 
Testing  in  Transformation  Roadmap,  which  calls  for  the  cre¬ 
ation  of  a  joint  test,  training  and  experimentation  infrastruc¬ 
ture  capable  of supporting  the  conduct  of each  in  a  joint  mission 
environment.  A  career  Navy  officer,  Crisp  concluded  his  23- 
year  naval  career  as  senior  military  assistant  to  the 
DOT&E/OSD  from  June  1998  to  June  2001,  where  he  was 
the  principal  military  advisor  to  the  director,  and  where  he 
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in  the  private  sector,  he  was  involved  in  program  management 
for  the  Defense  Information  Systems  Agency/Joint 
Interoperability  Test  Command  Joint  Distributed  Engineering 
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sible  for  program  development  and  coordination  with  OSD, 
Joint  Forces  Command  and  the  Services  in  developing  a  collab¬ 
orative,  distributed  engineering,  test  and  experimentation 
infrastructure  for  system-of-systems  interoperability  and  inte¬ 
gration  testing.  He  holds  a  master  of  science  degree  in  nation¬ 
al  security  and  strategic  studies  from  the  National  War 
College;  a  bachelor  of  science  degree  from  the  U.S.  Naval 
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